Monitoring Multiple Data Streams in Real Time

Online monitoring of data streams poses a challenge in many data-centric applications, such as telecommunications networks, traffic management, trend-related analysis, webclick streams, intrusion detection, and sensor networks. Mining techniques employed in these applications have to be efficient in terms of space usage and per-item processing time while providing a high quality of answers to (1) aggregate monitoring queries, such as finding surprising levels, i.e., “volatility” of a data stream, and detecting bursts, and to (2) similarity queries, such as detecting correlations and finding similar patterns. We propose a framework for summarizing a set of data streams, and for constructing a composite index structure in order to answer the above types of user queries. In our technique, features of streams are extracted incrementally on the fly at multiple resolutions, and inserted into a family of dynamic index structures. We demonstrate the effectiveness of our method over existing techniques through an extensive set of experiments. In answering aggregate queries, our false alarm rate is up to 400 times lower than current solutions. In the case of pattern analysis, our technique offers more than two times better accuracy while minimizing the space required for incremental computation. In detecting correlations, our technique performs up to 60 times Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the VLDB copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Very Large Data Base Endowment. To copy otherwise, or to republish, requires a fee and/or special permission from the Endowment. Proceedings of the 30th VLDB Conference, Toronto, Canada, 2004 better in response time, and up to 20 times better in terms of the quality of answers provided.